After completing this lesson, you’ll be able to:
Assessing the performance of a workspace such as this:
...requires a bit of time and attention, but knowing how a workspace performs helps to make informed decisions about how best to design (or redesign) it.
When testing for performance, ensure that only one instance of FME is running and that all other non-essential programs are closed. You'll also need to turn off data caching, which can affect the results. Finally, you must turn on log timestamping under Utilities > FME Options > Translation > Log Timestamp Information.
⭐ New for FME 2025.1: we've simplified the ability to write a CSV with some performance profiling information. You can access it under Run > Enable Performance Profiling. After running, FME will write a CSV to the same directory as the workspace with some performance timing information. Note that it provides times at the factory level, making interpretation tricky. The method outlined in this lesson will be more useful in most cases.
Improving a reader's efficiency requires estimating how well it works, yet it can take a lot of work to separate the reading speed in a workspace that also transforms data.
The key message that signifies the reading is complete is “Emptying Factory Pipeline.” Here, for example, the reading of the data finished after 26.4 seconds of processing (of course, the actual elapsed time might be longer if FME was waiting for a database or the file system to respond):
2024-03-29 11:48:54| 26.4| 0.0|INFORM|Emptying factory pipeline
However, sometimes, that message can be misleading. FME processes the data at the same time it is reading it, so that it won't read an entire data set before processing. To avoid this confusion, disable all the transformers and run only the readers:
With the transformers disabled, the message now appears much sooner:
2024-03-29 11:49:35| 1.2| 0.0|INFORM|Emptying factory pipeline
So, we can tell that reading the data only takes 1.2 seconds.
As with readers, you can only improve a writer's performance if you can first assess how well it is already performing. However, assessing the writing speed has the same complexity as reading: FME starts writing data as soon as it becomes available and doesn’t necessarily wait until processing is done.
So, how can we assess writer performance? Assessing a writer is different from assessing a reader. If you isolate the writers by disabling everything else, there won't be any data to write!
The easiest way is to disable the writer itself!
If the original result was this:
2024-03-29 11:54:14| 26.5| 0.0|INFORM|Translation was SUCCESSFUL with 19 warning(s) (148 feature(s) output) 2024-03-29 11:54:14| 26.5| 0.0|INFORM|FME Session Duration: 26.7 seconds. (CPU: 22.5s user, 4.0s system)
...and with the writer disabled, it is now this:
2024-03-29 11:56:48| 26.6| 0.0|INFORM|Translation was SUCCESSFUL with 19 warning(s) (0 feature(s) output) 2024-03-29 11:56:48| 26.6| 0.0|INFORM|FME Session Duration: 26.6 seconds. (CPU: 22.1s user, 4.4s system)
...then we can easily calculate that the writing process is taking merely 0.1 seconds.
Another method is a two-step process.
First, add a Recorder as the final transformer to store the data in FFS format as it is about to be written. Then, run the workspace.
Then, replace the Recorder with a Player transformer to re-read the preserved FFS data. Disable everything else up to that point and re-run the workspace.
The data is played back into the workspace and written to the output, measuring writer performance.
Assessing the time taken in transformation requires a two-step process.
First, disable writers and run the translation, taking note of the elapsed time:
Then disable the transformers, too, and rerun the workspace to calculate the time taken to read the data only:
The elapsed transformation time is the difference between the time spent reading the data and the time spent reading and transforming the data. In this case, 20.1 - 1.7 = 18.4 seconds spent transforming the data.
It's important not to add an Inspector or Logger transformer to see what happens to the output. This will slow the translation down and give you a false measure. You must also be sure to disable the actual writer and not just the feature types or connections to them.
The only writer helpful in this scenario is the Null format writer. This causes a writer to be present, but it does nothing except count features and discard them. The benefit is improved logging of record counts, but without any data having to be written.
Naturally, where a FeatureReader or FeatureWriter transformer is used, assessing reading and writing requires a different procedure:
It would be necessary to run the workspace twice, once with everything disabled after that point and once with everything disabled up to (and including) that point. Then, subtract the two run times to get the correct result.
